Object recognition method of autonomous driving device, and autonomous driving device

ABSTRACT

Disclosed is an object recognition method including: obtaining a first RGB image by using a camera; predicting at least one first region, in which an object is unrecognizable, in the first RGB image based on brightness information of the first RGB image; determining at least one second region, in which an object exists, from among the at least one first region, based on object information obtained through a dynamic vision sensor; obtaining an enhanced second RGB image by controlling photographic configuration information of the camera in relation to the at least one second region; and recognizing the object in the second RGB image.

TECHNICAL FIELD

The present disclosure relates to a method of recognizing a neighboringobject during autonomous driving and an autonomous driving device usingthe method.

BACKGROUND ART

As interest in autonomous vehicles increases, technologies that enableautonomous driving is attracting attention. In order for a vehicle tomove by itself without a driver's operation, (1) a technology forrecognizing the external environment of the vehicle, (2) a technologyfor synthesizing recognized information, determining an operation suchas acceleration, stop, and turning, and determining a driving route, and(3) a technology for controlling the movement of the vehicle by usingthe determined information are used. All these technologies have to beorganically combined to accomplish autonomous driving, but thetechnology for recognizing the external environment of a vehicle isgetting more and more important. This is because recognizing theexternal environment is the first element of autonomous driving, andfusion of electric, electronic, and information technologies is neededto recognize the external environment.

The technology for recognizing the external environment may be roughlyclassified into a sensor-based recognition technology and aconnection-based recognition technology. Sensors mounted on a vehiclefor autonomous driving include ultrasonic sensors, cameras, radars, andLIDAR sensors, and these sensors mounted on a vehicle, either alone ortogether with other sensors, recognize the external environment of thevehicle and the topography and provide information to a driver and thevehicle.

The connection-based recognition technology for autonomous drivinginclude V2X and precision positioning. V2X refers tovehicle-to-everything, which includes vehicle-to-vehicle (V2V) forcommunicating between vehicles, vehicle-to-infrastructure (V2I) forcommunicating with infrastructure, and vehicle-to-pedestrian (V2P) forcommunicating with pedestrians. V2X may refer to a wirelesscommunication technology that connects a traveling vehicle tosurrounding vehicles, transportation infrastructure, and nearbypedestrians. Information such as positions, distances, and speeds ofvehicles may be exchanged through an established communication network,and information such as surrounding traffic information and positions ofpedestrians may be provided to the vehicle.

DESCRIPTION OF EMBODIMENTS Solution to Problem

An embodiment relates to an object recognition method performed by anautonomous driving device, whereby a recognition rate of an externalobject is increased through use of a camera by adjusting photographicconfiguration information of the camera based on object informationdetected by a dynamic vision sensor.

According to an embodiment, a method, performed by an autonomous drivingdevice, of recognizing an object includes: obtaining a first RGB imageby using a camera arranged in the autonomous driving device; predictingat least one first region in which an object is unrecognizable in thefirst RGB image based on brightness information of the first RGB image;determining at least one second region in which an object exists fromamong the at least one first region based on object information obtainedthrough a dynamic vision sensor (DVS) arranged in the autonomous drivingdevice; obtaining an enhanced second RGB image by controllingphotographic configuration information of the camera in relation to theat least one second region; and recognizing the object in the second RGBimage.

According to an embodiment, an autonomous driving device includes: acamera; a dynamic vision sensor (DVS); and at least one processor,wherein the at least one processor is configured to: obtain a first RGBimage by using the camera; predict at least one first region in which anobject is unrecognizable in the first RGB image based on brightnessinformation of the first RGB image; determine at least one second regionin which an object exists from among the at least one first region basedon object information obtained through the dynamic vision sensor; obtainan enhanced second RGB image by controlling photographing configurationinformation of the camera in relation to the at least one second region;and recognize the object in the second RGB image.

According to an embodiment, a computer program product may store thereina program for: obtaining a first RGB image by using a camera; predictingat least one first region in which an object is unrecognizable in thefirst RGB image based on brightness information of the first RGB image;determining at least one second region in which an object exists fromamong the at least one first region based on object information obtainedthrough a dynamic vision sensor (DVS); obtaining an enhanced second RGBimage by controlling photographing configuration information of thecamera in relation to the at least one second region; and recognizingthe object in the second RGB image.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a diagram for explaining an autonomous driving device,according to an embodiment.

FIG. 2 is a flowchart for explaining an object recognition method usedby an autonomous driving device, according to an embodiment.

FIG. 3 is a diagram for explaining an object-unrecognizable region and aregion of interest determined in an RGB image, according to anembodiment.

FIG. 4 is a diagram for explaining photographic configurationinformation of a camera, according to an embodiment.

FIG. 5 is a flowchart for explaining a method of recognizing an objectby using a histogram, according to an embodiment.

FIG. 6 is a diagram for explaining an operation of determining whetheran object-unrecognizable region exists in an RGB image by using ahistogram, according to an embodiment.

FIG. 7 is a diagram for explaining an operation of determining anobject-unrecognizable region in an RGB image, according to anembodiment.

FIG. 8 is a flowchart for explaining a method of recognizing an objectby using an artificial intelligence model, according to an embodiment.

FIG. 9 is a diagram for explaining an operation of applying an RGB imageand a DVS image to an AI processor, according to an embodiment.

FIG. 10 is a diagram for explaining an operation performed by anautonomous driving device to obtain an enhanced RGB image, according toan embodiment.

FIG. 11 is a diagram for explaining an operation performed by anautonomous driving device to obtain an enhanced RGB image when enteringa tunnel, according to an embodiment.

FIG. 12 is a diagram for explaining an operation of controllingphotographic configuration information of a camera when anobject-unrecognizable region due to a backlight exists in an RGB image,according to an embodiment.

FIG. 13 is a flowchart for explaining a method of controllingphotographic configuration information of a camera according topriorities of a plurality of regions of interest, according to anembodiment.

FIG. 14 is a diagram for explaining priorities of a plurality of regionsof interest, according to an embodiment.

FIG. 15 is a flowchart for explaining a method performed by anautonomous driving device to track an object, according to anembodiment.

FIG. 16 is a diagram for explaining an operation performed by anautonomous driving device to recognize and track a new object detectedby a dynamic vision sensor, by using a camera, according to anembodiment.

FIG. 17 is a block diagram for explaining a configuration of anautonomous driving device, according to an embodiment.

FIG. 18 is a block diagram of a processor, according to an embodiment.

FIG. 19 is a block diagram of a data learner, according to anembodiment.

FIG. 20 is a block diagram of a data recognizer, according to anembodiment

FIG. 21 is a diagram illustrating an example in which an autonomousdriving device and a server interoperate to learn and recognize data,according to an embodiment.

MODE OF DISCLOSURE

The terms used in the present specification will be briefly describedand embodiments of the present disclosure will be described in detail.

The terms used in the present disclosure are selected from among commonterms that are currently widely used in consideration of their functionin the present disclosure. However, the terms may be different accordingto an intention of one of ordinary skill in the art, a precedent, or theadvent of new technology. Also, in particular cases, the terms arediscretionally selected by the applicant of the present disclosure, inwhich case, the meaning of those terms will be described in detail inthe corresponding part of the detailed description. Therefore, the termsused in the present disclosure are not merely designations of the terms,but the terms are defined based on the meaning of the terms and contentthroughout the present disclosure.

Throughout the specification, when a part “includes” an element, it isto be understood that the part additionally includes other elementsrather than excluding other elements as long as there is no particularopposing recitation. Also, the terms described in the specification,such as “ . . . er (or)”, “ . . . unit”, “ . . . module”, etc., denote aunit that performs at least one function or operation, which may beimplemented as hardware or software or a combination thereof.

Hereinafter, embodiments of the present disclosure will now be describedin detail with reference to the accompanying drawings for one of skillin the art to be able to perform the present disclosure without anydifficulty. The present disclosure may, however, be embodied in manydifferent forms and should not be construed as being limited to theembodiments of the present disclosure set forth herein. Also, parts inthe drawings unrelated to the detailed description are omitted to ensureclarity of the present disclosure, and like reference numerals in thedrawings denote like elements.

FIG. 1 is a diagram for explaining an autonomous driving device,according to an embodiment.

The autonomous driving device 100 according to an embodiment may referto a device capable of autonomous driving without depending on a controlcommand input from the outside, and may include, for example, anautonomous driving vehicle, an autonomous flying device (e.g., a droneor an unmanned flying device), an autonomous driving robot (e.g., acleaning robot or a disaster rescue robot), etc., but is not limitedthereto. Hereinafter, for convenience of descriptions, a case where theautonomous driving device 100 is the autonomous driving vehicle will bedescribed as an example.

According to an embodiment, the autonomous driving device 100 mayinclude a camera 101, a dynamic vision sensor 102, and a processor 120,but is not limited thereto. For example, the autonomous driving device100 may further include a lidar sensor, a radar sensor, an inertialsensor (inertial measurement unit (IMU)), an ultrasonic sensor, aninfrared sensor, a position sensor (e.g., a global positioning system(GPS) module), a geomagnetic sensor, an acceleration sensor, a gyroscopesensor, etc. According to another embodiment, the autonomous drivingdevice 100 may further include a communicator (e.g., a Bluetoothcommunicator, a Bluetooth low energy (BLE) communicator, a near-fieldcommunication (NFC) communicator, a Zigbee communicator, an ultra-wideband (UWB) communicator, and a mobile communicator), a driving unit (apower supply, a propelling unit, a traveling unit, and a peripheraldevice unit), an outputter, and a storage unit. The configuration of theautonomous driving device 100 will be described in detail with referenceto FIG. 17 .

The camera 101 according to an embodiment may recognize at least oneobject that is present within a certain distance from the autonomousdriving device 100. Here, one or more cameras 101 for recognizing theobject may be provided. For example, the camera 101 may be at least oneof a front camera, a rear camera, and a side camera, and the camera 101may be a stereo camera or an around-view camera.

Meanwhile, the object captured by the camera 101 may include a staticenvironment element (e.g., a lane, a drivable road, a traffic sign, atraffic light, a tunnel, a bridge, a street tree, etc.) and a dynamicenvironment element (e.g., a vehicle, a pedestrian, a motorcycle, etc.),but is not limited thereto. For example, the object captured by thecamera 101 may include features (e.g., a feature point and a featureline) that may be applied to a position recognition technology (e.g.,simultaneous localization and mapping (SLAM) or visual inertial odometry(VIO)).

However, because a dynamic range of a typical camera 101 is not high, itis difficult for the camera 101 to capture an object in a very darkplace or a very bright place. For example, when entering a tunnel (shownin 100-1), the camera 101 arranged in the autonomous driving device 100may have difficulty in capturing an object in a dark region 10 in thetunnel. Also, when exiting the tunnel (shown in 100-2), the camera 101arranged in the autonomous driving device 100 may have difficulty incapturing an object in a bright region 20 outside the tunnel, and mayalso have difficulty in capturing an object in a region 30 beingbrightly illuminated by a backlight (shown in 100-3). Furthermore, whenthe autonomous driving device 100 is passing a zone with extreme changesin illumination, or a shadowed zone, is moving at a high speed at night,or an object with a color similar to that of the background appears, itis difficult for the camera 101 to clearly capture the object.

Therefore, for safe driving of the autonomous driving device 100, it isnecessary to increase an object recognition rate of the camera 101 in alow-illumination environment or in the presence of a backlight. Forexample, according to an embodiment, the autonomous driving device 100may increase the object recognition rate of the camera 101 bycontrolling photographic configuration information of the camera 101 byusing information detected by the dynamic vision sensor 102.

The dynamic vision sensor 102 is an event-based camera that captures avision change at a high speed, and is a sensor that may obtain imagedata of a moving object. For example, the dynamic vision sensor 102 maytransmit the image data to the processor 120 only when a local changedue to a motion in a pixel unit occurs. That is, the dynamic visionsensor 102 may transmit the image data to the processor 120 when amotion event occurs.

The dynamic vision sensor 102 may solve a problem that a typical visionrecognition system is vulnerable to a rapid motion. Because the dynamicvision sensor 102 receives data on a per-pixel basis rather than a framebasis, a blur phenomenon may be overcome.

In addition, the dynamic vision sensor 102 may have a resolution inmicroseconds. In other words, the dynamic vision sensor 102 may have atemporal resolution (e.g., a super high-speed frame>1K FPS) better thana super high-speed camera that shoots thousands of frames per second. Inaddition, the dynamic vision sensor 102 has dramatically reduced powerconsumption and data storage requirements, resulting in a dramaticincrease in a dynamic range (a range of brightness identifiable by asensor). Accordingly, the dynamic vision sensor 102 may detect themotion of the object when only a slight amount of light is present evenin a dark place.

According to an embodiment, the dynamic vision sensor 102 may be closeto the camera 101. In addition, a field of view (FOV) of the dynamicvision sensor 102 or a pose of the dynamic vision sensor 102 may beadjusted such that the dynamic vision sensor 102 obtain an image of aregion similar to that of the image being captured by the camera 101.According to an embodiment, a frame rate of the dynamic vision sensor102 may be set to be the same as or similar to that of the camera 101,but is not limited thereto.

According to an embodiment, while the autonomous driving device 100 isdriving, the dynamic vision sensor 102 arranged in the autonomousdriving device 100 may detect a local change in pixel units, and maytransmit information about the detected local change to the processor120. In this case, because the dynamic vision sensor 102 has a dynamicrange wider than that of the camera 101, the processor 120 may receiveinformation about an object, that is not captured by the camera 101,from the dynamic vision sensor 102. In this case, the processor 120 maycontrol the photographic configuration information of the camera 101such that the object, that is not captured by the camera 101 but isdetected by the dynamic vision sensor 102, may be captured by the camera101 as well. According to an embodiment, the processor 120 may include ageneral image signal processor (ISP) or an artificial intelligenceprocessor (AI processor).

Hereinafter, a method, performed by the processor 120 of the autonomousdriving device 100, of increasing the object recognition rate of thecamera 101 by controlling the photographic configuration information ofthe camera 101, by using information detected by the dynamic visionsensor 102 will be described in detail with reference to FIG. 2 .

FIG. 2 is a flowchart for explaining an object recognition method of anautonomous driving device, according to an embodiment.

In operation S210, the autonomous driving device 100 may obtain a firstRGB image by using the camera 101.

According to an embodiment, the first RGB image is an image forrecognizing at least one object around the autonomous driving device100, and may be composed of at least one frame. For example, in a caseof the first RGB image being a still image, the first RGB image may becomposed of a single frame, while, in a case of the first RGB imagebeing a moving image, the first RGB image may be composed of a pluralityof frames.

According to an embodiment, the autonomous driving device 100 may obtainthe first RGB image by using the camera 101 while driving, or may obtainthe first RGB image by using the camera 101 after the autonomous drivingdevice 100 is parked or stopped.

In operation S220, the autonomous driving device 100 may predict atleast one first region in which an object is unrecognizable, from thefirst RGB image based on brightness information of the first RGB image.Here, the brightness information may be information indicating abrightness level of each pixel in the first RGB image. The brightnessinformation may include a brightness value of each pixel, informationabout a region that is darker than a first reference brightness value,and information about a region that is brighter than a second referencebrightness value, but is not limited thereto.

According to an embodiment, the autonomous driving device 100 maydetermine whether a probability that an object-unrecognizable regionexists in the first RGB image exceeds a threshold value. Theobject-unrecognizable region may refer to a region in which thebrightness values are out of a threshold range (e.g., 50 to 200) (e.g.,a considerably dark or bright region). For example, the autonomousdriving device 100 may determine the probability that theobject-unrecognizable region exists in the first RGB image by using ahistogram of the first RGB image. In a case where the distribution ofthe histogram of the first RGB image is biased to 0 or 255, theautonomous driving device 100 may determine that the probability thatthe object-unrecognizable region exists in the first RGB image is high.An operation of, by the autonomous driving device 100, using thehistogram will be described in detail with reference to FIG. 5 .

According to an embodiment, the autonomous driving device 100 maydetermine the probability that the object-unrecognizable region existsin the first RGB image, based on a difference between the first RGBimage and a dynamic vision sensor (DVS) image obtained through thedynamic vision sensor 102. For example, the greater a difference betweenedge information (or intensity information) detected from the DVS imageand edge information (or intensity information) detected from the firstRGB image is, the higher the probability that the object-unrecognizableregion exists in the first RGB image the autonomous driving device 100may determine.

According to an embodiment, the autonomous driving device 100 maydetermine the probability that the object-unrecognizable region existsin the first RGB image by using an artificial intelligence model thathas been trained based on a plurality of RGB images. An operation of, bythe autonomous driving device 100, using the artificial intelligencemodel will be described in detail with reference to FIG. 8 .

In a case where the autonomous driving device 100 has determined thatthe probability that the object-unrecognizable region exists in thefirst RGB image is greater than the threshold value, the autonomousdriving device 100 may predict the at least one first region in which anobject is unrecognizable, from the first RGB image, by using thebrightness information of the first RGB image. Here, the at least onefirst region in which an object is unrecognizable may be a region inwhich a degree indicating how likely an object is unrecognizable exceedsa threshold value. For example, the autonomous driving device 100 maydefine, as the at least one first region in which an object isunrecognizable, a region in which the brightness values are out of thethreshold range in the first RGB image (e.g., a considerably dark orbright region). Accordingly, only a few features may be detected in theat least one first region in which an object is unrecognizable.Hereinafter, for convenience of description, the at least one firstregion in which an object is unrecognizable may be expressed as theobject-unrecognizable region.

In operation S230, the autonomous driving device 100 may determine atleast one second region in which an object exists, from among the atleast one first region, based on object information obtained through thedynamic vision sensor 102 arranged in the autonomous driving device 100.Hereinafter, for convenience of description, the at least one secondregion may be expressed as a region of interest.

According to an embodiment, the autonomous driving device 100 may obtainthe object information through the dynamic vision sensor 102. The objectinformation is information about the object detected by the dynamicvision sensor 102, and may include at least one of the DVS image andposition information of at least one object detected from the DVS image,but is not limited thereto.

According to an embodiment, the autonomous driving device 100 maycompare the DVS image with the first RGB image to define, as the regionof interest, a region, from among the at least one first region of thefirst RGB image, in which a probability that an object exists is greaterthan the threshold value (e.g., 98%). For example, the region ofinterest may be a region, the DVS image of which has a large amount offeature information, while the first RGB image of which has a smallamount of feature information.

The object-unrecognizable region and the region of interest will bedescribed with reference to FIG. 3 . Referring to 300-1 of FIG. 3 , theautonomous driving device 100 may obtain an RGB image 310 while passingthrough the tunnel. In this case, on the RGB image 310, a tunnel exitregion may appear bright while a region inside the tunnel may appeardark. The autonomous driving device 100 may analyze the RGB image 310 byusing the histogram of the RGB image 310 or by using the artificialintelligence model. As a result of the analysis, the autonomous drivingdevice 100 may define regions 311, 312, 313, 314, 315 (tunnel exitregion), 316 and 317, in which the brightness values are out of thethreshold range, as the object-unrecognizable regions. In this case, theregions 311, 312, 313, 316, and 317 may be the regions inside the tunneland are considerably dark, while the regions 314 and 315 may be regionsthat are considerably bright due to light incident from an exit of thetunnel.

Referring to 300-2 of FIG. 3 , the autonomous driving device 100 maycompare the regions 311 to 317 of the RGB image 310 with correspondingregions of the DVS image 320, respectively. Here, as the tunnel exitregion 315 is bright, an object may not be captured in the RGB image310, whereas the object may be detected in a corresponding region 321 ofthe DVS image. Because the dynamic vision sensor 102 has a dynamic rangewider than that of the camera 101, the dynamic vision sensor 102 maydetect an object in a bright region.

Accordingly, the autonomous driving device 100 may define, as the regionof interest, a region in which an object is detected by the dynamicvision sensor 102 (e.g., the tunnel exit region 315), but in which theobject is not captured by the camera 101.

In operation S240, the autonomous driving device 100 may obtain anenhanced second RGB image corresponding to the at least one secondregion (for convenience of description, referred to as the region ofinterest), by controlling the photographic configuration information ofthe camera 101. Here, the photographic configuration information of thecamera 101 may include exposure information, focus information, whitebalance information, or mode information, but is not limited thereto.Also, the enhanced second RGB image may refer to an image having abrightness adjusted to enable object detection in a region correspondingto the second region of the first RGB image.

According to an embodiment, the autonomous driving device 100 may checkthe current photographic configuration information of the camera 101 andcontrol the photographic configuration information such that thebrightness of the region of interest may be adjusted. For example, theautonomous driving device 100 may control at least one of exposure,focus, and white balance with respect to the region of interest. Inparticular, the autonomous driving device 100 may control an exposurevalue with respect to the region of interest by adjusting at least oneof a gain, aperture, and exposure time of the camera 101. For example,in a case where the region of interest is a dark region, the autonomousdriving device 100 may appropriately adjust the gain, the aperture, andthe exposure time to control the region of interest to appear brighterin the enhanced second RGB image. In contrast, in a case where theregion of interest is a bright region, the autonomous driving device 100may appropriately adjust the gain, the aperture, and the exposure timeto control the region of interest to appear darker in the enhancedsecond RGB image.

Referring to FIG. 4 , the aperture 410 refers to a hole of a lensthrough which light passes. As the aperture 410 is closed (right) toincrease a depth, an image where a near region and a far region arefocused is output, whereas, as the aperture 410 is opened (left) toreduce the depth, an image where a subject and a background areseparated from each other, referred to as out of focus, is output. As ashutter speed 420 increases (left), an image where a fast moving objectappears frozen is output, whereas, as the shutter speed 420 decreases(right), a blurred image is output. As an ISO sensitivity 430 decreases(left), an image with small noise is output. As the ISO sensitivity 430increases (right), noise increases and an image with no shake may betaken even in a dark environment.

As the ISO sensitivity 430 decreases (left), a contrast increases. Incontrast, as the ISO sensitivity 430 increases, the contrast is reduced,and thus a blunt image is taken. In a case of the ISO sensitivity 430being low, film grains are thin and lead to a sharp image, whereas, in acase of the ISO sensitivity 430 being high, the film grains are thickand lead to a rough image.

Therefore, according to an embodiment, in a case where the region ofinterest is dark, the autonomous driving device 100 may increase thesensitivity 430 of the camera 101 or may control the shutter speed 420to be decreased. In contrast, in a case where the region of interest isbright, the autonomous driving device 100 may decrease the sensitivity430 of the camera 101.

Meanwhile, according to an embodiment, in a case of the region ofinterest being brightly illuminated by a backlight, the autonomousdriving device 100 may change a metering mode, for example, to any oneof evaluative metering, partial metering, center-weighted averagemetering, or spot metering, or may change an autofocus point (AF point).For example, in a case of the region of interest being a bright region,the autonomous driving device 100 may obtain the second RGB image whichis entirely dark by moving the AF point to the region of interest.

In addition, according to an embodiment, the autonomous driving device100 may select a wide dynamic range (WDR) function. The wide dynamicrange (WDR) is a technology for enabling both bright regions and darkregions of an image to clearly appear. By this technology, a high-speedshutter image signal for the bright region and a low-speed shutter imagesignal for the dark region are merged into an image, and thus a problemdue to a backlight may be resolved to generate a clear image.

According to an embodiment, the AI processor of the autonomous drivingdevice 100 may control the photographic configuration information of thecamera 101 by using the artificial intelligence model that has trainedto control the photographic configuration information. An operation of,by the autonomous driving device 100, controlling the photographicconfiguration information of the camera 101 by using the artificialintelligence model will be described in detail with reference to FIG. 8.

In operation S250, the autonomous driving device 100 may recognize theobject in the second RGB image.

According to an embodiment, the autonomous driving device 100 mayextract at least one feature that constitutes the object, from a regionof interest of the second RGB image. The region of interest of thesecond RGB image may correspond to the region of interest of the firstRGB image. The autonomous driving device 100 may recognize the object inthe region of interest of the second RGB image by using the at least oneextracted feature. According to an embodiment, the recognizing of theobject may include determining a type of the object.

According to an embodiment, the autonomous driving device 100 mayrecognize the object in the region of interest of the second RGB imagethat corresponds to the region of interest of the first RGB image, byusing template information or the artificial intelligence model. Forexample, the autonomous driving device 100 may determine the type of theobject by analyzing the second RGB image obtained through the camera101. For example, in a case of the object being an external vehicle, theautonomous driving device 100 may detect an outline of the externalvehicle included in the second RGB image, as the feature. The autonomousdriving device 100 may compare the detected outline of the externalvehicle with a predefined template to detect a type of the externalvehicle, a name of the external vehicle, etc. For example, in a case ofthe outline of the external vehicle being similar to a template of abus, the autonomous driving device 100 may recognize the externalvehicle as a bus. In addition, because a typical bus is large and heavy,the autonomous driving device 100 may define the type of the externalvehicle as a large vehicle.

According to an embodiment, the autonomous driving device 100 mayrecognize the object in the region of interest of the second RGB imageby using a precision map. Here, the precision map may include not onlyroad information necessary for the vehicle to travel but also a mapwhich is much more precise than an existing map and has an error of, forexample, 10-20 cm or less from an actual road. For example, theautonomous driving device 100 may call a precision map of surroundingsof the autonomous driving device 100. The autonomous driving device 100may compare the second RGB image with the called precision map torecognize a static object in the region of interest of the second RGBimage. For example, the autonomous driving device 100 may recognize thatthe object is a lane, a stop line, a road sign, a road structure, etc.,by comparing the features extracted from the second RGB image with theprecision map.

Meanwhile, the autonomous driving device 100 may identify a currentposition of the recognized object (e.g., absolute position), a lane inwhich the external vehicle is driving (e.g., first lane) in a case ofthe recognized object being the external vehicle, etc., by using theprecision map.

According to an embodiment, in a case of the recognized object being adynamic object (e.g., an external vehicle), the autonomous drivingdevice 100 may track the recognized object by using the camera 101.Object tracking refers to tracking changes in an object by usingsimilarities between characteristic information such as sizes, colors,shapes, or contours of the same objects in a series of image frames.

According to an embodiment, the dynamic vision sensor 102 may detect anew object appearing around the autonomous driving device 100 earlierthan the camera 101 does. Therefore, according to an embodiment, in acase where the new object has been detected by the dynamic vision sensor102, the autonomous driving device 100 may determine, based on aposition where the new object is detected, a candidate region in which apossibility of recognizing the new object on the RGB image of the camera101 is greater than a threshold value. The autonomous driving device 100may recognize and track the new object on the RGB image, by performingimage processing on the candidate region. In this case, the autonomousdriving device 100 may rapidly recognize the new object by performingthe image processing on only the candidate region, rather than on theentirety of the RGB image, in order to capture the new object by usingthe camera 101. An operation of, by the autonomous driving device 100,recognizing and tracking the object will be described in detail withreference to FIG. 15 .

According to an embodiment, in a case of the second RGB image beingcomposed of a plurality of frames, the autonomous driving device 100 mayobtain position information of the autonomous driving device 100 bytracking a feature included in the object recognized from each of theplurality of frames. For example, the autonomous driving device 100 mayuse the feature included in the object recognized from the second RGBimage, as a feature to be applied to visual odometry (e.g., visualodometry using VIO or a stereo camera). Here, the visual odometry is atechnology for predicting a position change of a mobile device by usinga difference between a previous frame and a current frame.

According to an embodiment, because calculation of changes in theprevious frame and the current frame with respect to all pixels requiresa considerably high amount of calculation, the autonomous driving device100 may extract features such as lines or corners that may represent achange in a scene from each frame and may match the extracted features.

According to an embodiment, the autonomous driving device 100 maygenerate a motion vector from which a change in a position of a featurepoint on the scene may be predicted, by matching the feature pointextracted from the previous frame, in the current frame. Because themotion vector represents an image change in a two-dimensional space (x,y), the autonomous driving device 100 may convert the motion vector intocoordinates in a three-dimensional space (x, y, z) by adding distanceinformation (depth) from the stereo camera or distance information fromthe inertial sensor (IMU). The autonomous driving device 100 maycalculate a three-dimensional motion vector that represents an amount ofchanges in an actual space by using three-dimensional coordinatescorresponding to the feature point in the previous frame andthree-dimensional coordinates corresponding to the feature point in thecurrent frame, from a set of the matched feature points. The autonomousdriving device 100 may recognize a current position of the autonomousdriving device 100 by using the three-dimensional motion vector.

In an outdoor environment, because textures of roads may be neitheruniform nor flat, it is difficult to use position recognition using anencoder, and, in a case of a global positioning system (GPS), signalsmay not be received when surrounded by an artificial structure such as atunnel or a building, and it is difficult to use an inertial navigationsystem (INS) with six degrees of freedom because of its considerablyexpensive price. Therefore, according to an embodiment, a position ofthe autonomous driving device 100 may be recognized by using the featureextracted from the second RGB image, and thus disadvantages of the GPSand the INS may be mitigated.

According to an embodiment, the autonomous driving device 100 maygenerate a map based on position information recognized through thevisual odometry.

According to an embodiment, the autonomous driving device 100 maydetermine a route of the autonomous driving device 100 based oninformation about the object recognized from the second RGB image. Forexample, in a case where the object recognized from the second RGB imageis an obstacle, the autonomous driving device 100 may plan a motion foravoiding the obstacle. For example, the autonomous driving device 100may change a lane or decrease its speed. In addition, in a case wherethe object recognized from the second RGB image is a traffic lightindicating a stop sign, the autonomous driving device 100 may plan theroute for stopping in front of a stop line.

Therefore, according to an embodiment, the autonomous driving device 100may increase the object recognition rate of the camera 101 even in anenvironment with extreme changes in illumination, by controlling thephotographic configuration information of the camera 101 based on theinformation detected by the dynamic vision sensor 102. In addition, asthe object recognition rate of the camera 101 is increased, a currentposition recognition rate, precision in planning the route, and anobject tracking rate may be improved.

Hereinafter, an operation of, by the autonomous driving device 100,using the histogram will be described in detail with reference to FIG. 5.

FIG. 5 is a flowchart for explaining a method of recognizing an objectby using a histogram, according to an embodiment.

In operation S500, the autonomous driving device 100 may obtain thefirst RGB image by using the camera 101.

Operation S500 corresponds to operation S210 of FIG. 2 , andaccordingly, its detailed description will be omitted.

In operation S510, the autonomous driving device 100 may analyze thefirst RGB image to obtain the histogram of the first RGB image. Thehistogram represents a brightness distribution of an image in a graph.For example, the histogram may have a range of brightness values of 0 to255, and frequencies (the number of pixels) of each brightness value maybe represented as heights of rectangles. That is, a horizontal axis ofthe histogram may represent the brightness values, and a vertical axisof the histogram may represent numbers of pixels. Accordingly, in a caseof the first RGB image being generally dark, a histogram having adistribution biased to 0 may be obtained, while, in a case of the firstRGB image being generally bright, a histogram having a distributionbiased to 255 may be obtained.

In operation S520, the autonomous driving device 100 may determinewhether the object-unrecognizable region exists in the first RGB imageby using the histogram of the first RGB image.

According to an embodiment, the autonomous driving device 100 maydetermine that the object-unrecognizable region exists in a case wherethe distribution of the histogram is not uniform and biased leftward orrightward.

For example, referring to FIG. 6 , a brightness distribution of a firsthistogram 610 is not uniform, and biased leftward and rightward. In thiscase, the autonomous driving device 100 may determine that theprobability that the object-unrecognizable region exists in the firstRGB image is high because the first RGB image may be composed of mostlybright regions and dark regions. On the other hand, a brightnessdistribution of a second histogram 620 is uniform, from 0 to 255, theautonomous driving device 100 may determine that the probability thatthe object-unrecognizable region exists in the first RGB image is low.

In operation S530, in a case where the autonomous driving device 100 hasdetermined that the object-unrecognizable region does not exist, theautonomous driving device 100 may not modify the photographicconfiguration information of the camera 101. That is, the autonomousdriving device 100 may continuously obtain RGB images based on thecurrent photographic configuration information of the camera 101. Inoperation S535, because the object-unrecognizable region does not existin the first RGB image, the autonomous driving device 100 may recognizethe object from the first RGB image.

In operation S540, in a case where the autonomous driving device 100 hasdetermined that the object-unrecognizable region exists, the autonomousdriving device 100 may predict the at least one first region(object-unrecognizable region) in which an object is unrecognizable,from the first RGB image. Here, the at least one first region in whichan object is unrecognizable may be a region(s) in which the brightnessvalues are out of the threshold range.

When the first RGB image is converted into the histogram, all spatialinformation of the first RGB image is lost. That is, the histogramindicates the number of pixels having each brightness value, but doesnot provide any information about where the pixels are located.Therefore, the autonomous driving device 100 may determine a thresholdvalue by using the histogram, in order to identify theobject-unrecognizable region in the first RGB image. For example, theautonomous driving device 100 may analyze the histogram to determine afirst reference value for detecting a region in which an object isunrecognizable due to the region's darkness, or a second reference valuefor detecting a region in which an object is unrecognizable due to theregion's brightness.

For example, referring to FIG. 7 , the histogram 700 of the first RGBimage may be divided into a group of pixels having brightness valuesbetween 0 and 70 and another group of pixels having brightness valuesbetween 240 and 250. Therefore, the autonomous driving device 100 maydefine the first reference value 701 as ‘80’, for detecting the darkregion 710 in which an object is unrecognizable due to the region'sdarkness. In this case, the autonomous driving device 100 may define thedark region 710 in which an object is unrecognizable due to the region'sdarkness by representing a pixel having a brightness value less than 80as ‘1’ and representing a pixel having a brightness value greater thanor equal to 80 as ‘0’. In addition, the autonomous driving device 100may define the second reference value 702 as ‘230’, for detecting thebright region 720 in which an object is unrecognizable due to theregion's brightness. In this case, the autonomous driving device 100 maydefine the bright region 720 in which an object is unrecognizable due tothe region's brightness by representing a pixel having a brightnessvalue greater than 230 as ‘1’ and representing a pixel having abrightness value less than or equal to 230 as ‘0’.

Although FIG. 7 illustrates a case where the first reference value 701is different from the second reference value 702, the present disclosureis not limited thereto. According to an embodiment, the first referencevalue 701 and the second reference value 702 may be the same value. Forexample, the autonomous driving device 100 may define both the firstreference value 701 and the second reference value 702 as ‘150’. In thiscase, the autonomous driving device 100 may define the dark region 710by representing a pixel having a brightness value less than 150 as ‘1’,and may define the bright region 720 by representing a pixel having abrightness value greater than 150 as ‘1’.

In operation S550, the autonomous driving device 100 may compare theobject information of the DVS image obtained through the dynamic visionsensor 102 with the object information of the first RGB image, todetermine the at least one second region (region of interest) in whichan object exists, from among the at least one first region.

For example, the autonomous driving device 100 may define, as the regionof interest, a region, the DVS image of which has a large amount offeature information, while the first RGB image of which has a smallamount of feature information.

Operation S550 corresponds to operation S230 of FIG. 2 , andaccordingly, its detailed description will be omitted.

In operation S560, the autonomous driving device 100 may control thephotographic configuration information of the camera 101 in relation tothe at least one second region (region of interest). In operation S570,the autonomous driving device 100 may obtain the enhanced second RGBimage from the camera 101 based on the modified photographicconfiguration information.

According to an embodiment, the autonomous driving device 100 may checkthe current photographic configuration information of the camera 101 andcontrol the photographic configuration information such that thebrightness of the region of interest may be modified. For example, theautonomous driving device 100 may control at least one of exposure,focus, and white balance with respect to the region of interest. Inparticular, the autonomous driving device 100 may control the exposurevalue with respect to the region of interest by adjusting at least oneof the gain, aperture, and exposure time of the camera 101. For example,in a case where the region of interest is a dark region, the autonomousdriving device 100 may appropriately adjust the gain, the aperture, andthe exposure time to control the region of interest to appear brighterin the enhanced second RGB image. In contrast, in a case where theregion of interest is a bright region, the autonomous driving device 100may appropriately adjust the gain, the aperture, and the exposure timeto control the region of interest to appear darker in the enhancedsecond RGB image.

Operations S560 and S570 correspond to operation S240 of FIG. 2 , theirdetailed description will be omitted.

In operation S580, in a case where no object has not been recognizedfrom the region of interest (a region(s) corresponding to the at leastone second region of the first RGB image) of the second RGB image, theautonomous driving device 100 may control the photographic configurationinformation of the camera 101 in relation to the at least one secondregion (region of interest) again. For example, in a case of the regionof interest being a dark region, the autonomous driving device 100 mayfurther increase the gain of the camera 101 to capture the region ofinterest appearing brighter.

In operation S590, the autonomous driving device 100 may obtain theposition information of the autonomous driving device 100 by using arecognized object.

According to an embodiment, the autonomous driving device 100 may use afeature included in the recognized object as a feature to be applied toa VIO technology. For example, the autonomous driving device 100 mayextract the features from the region of interest of the current frameand the region of interest of the previous frame of the second RGBimage, respectively, and may predict a change in the position in atwo-dimensional space by matching the extracted features. The autonomousdriving device 100 may obtain information of an actual travel distanceby using the stereo camera or the inertial sensor. The autonomousdriving device 100 may estimate an amount of three-dimensional positionchanges by using the predicted position change and the distanceinformation obtained through the stereo camera or the inertial sensor.The autonomous driving device 100 may generate a three-dimensional mapof surroundings of the autonomous driving device 100 by using the amountof the three-dimensional position changes.

Hereinafter, an operation of, by the autonomous driving device 100,using the artificial intelligence model instead of the histogram will bedescribed in detail with reference to FIG. 8 .

FIG. 8 is a flowchart for explaining a method of recognizing an objectby using an artificial intelligence model, according to an embodiment.

In operation S810, the autonomous driving device 100 may obtain thefirst RGB image by using the camera 101.

Operation S810 corresponds to operation S210 of FIG. 2 , andaccordingly, its detailed description will be omitted.

In operation S820, the autonomous driving device 100 may determinewhether the object-unrecognizable region exists in the first RGB imageby using a first artificial intelligence model. According to anembodiment, the first artificial intelligence model is a neural networkmodel that learns from RGB images, and may be a model that has beentrained to determine an object-unrecognizable region in RGB images.According to an embodiment, the first artificial intelligence model maybe trained based on RGB images captured on a route through which theautonomous driving device 100 frequently travels.

According to an embodiment, when the first RGB image is input, the firstartificial intelligence model may identify dark regions and brightregions in the first RGB image. In this case, the first artificialintelligence model may determine whether the object-unrecognizableregion exists in the first RGB image by considering distributions of thedark regions and the bright regions. For example, in a case where thefirst RGB image has an irregular brightness distribution and isconsiderably dark or bright, the first artificial intelligence model maydetermine that the object-unrecognizable region exists in the first RGBimage.

In addition, according to an embodiment, in a case of surroundingsituation information (context information) being received, the firstartificial intelligence model may determine whether theobject-unrecognizable region exists in the first RGB image byconsidering the surrounding situation information. For example, when theautonomous driving device 100 is entering or passing through a tunnel,the first artificial intelligence model may determine that theprobability that the object-unrecognizable region exists in the firstRGB image is high. In addition, when the current position of theautonomous driving device 100 on a current driving route is where abacklight occurs, the first artificial intelligence model may determinethat the probability that the object-unrecognizable region exists in thefirst RGB image is high.

In operation S830, in a case where it has been determined that theobject-unrecognizable region does not exist, the autonomous drivingdevice 100 may not modify the photographic configuration information ofthe camera 101. That is, the autonomous driving device 100 maycontinuously obtain RGB images based on the current photographicconfiguration information of the camera 101. In operation S835, becausethe object-unrecognizable region does not exist in the first RGB image,the autonomous driving device 100 may recognize the object from thefirst RGB image.

In operation S840, in a case where it has been determined that theobject-unrecognizable region exists, the at least one first region inwhich an object is unrecognizable (object-unrecognizable region) may bepredicted from the first RGB image, by using the first artificialintelligence model.

For example, in a case where the autonomous driving device 100 appliesthe first RGB image to the first artificial intelligence model, thefirst artificial intelligence model may define, as theobject-unrecognizable region, a region having brightness values lowerthan the first reference value (e.g., 100) or a region having brightnessvalues higher than the second reference value (e.g., 150) in the firstRGB image.

In operation S850, the autonomous driving device 100 may determine theat least one second region in which an object exists, from among the atleast one first region, by applying the DVS image and the first RGBimage to a second artificial intelligence model.

According to an embodiment, the second artificial intelligence model maybe a model that learns from DVS images and RGB images with respect tothe same scene. The second artificial intelligence model may be a neuralnetwork model that compares a DVS image with an RGB image, and predictsa region where an object exists, from among object-unrecognizableregions of the RGB image.

According to an embodiment, the second artificial intelligence model maybe separated from or integrated with the first artificial intelligencemodel for determining the object-unrecognizable region in the RGB image.

Referring to FIG. 9 , the autonomous driving device 100 may obtain theRGB image 910 through the camera 101 while passing through a tunnel, andmay obtain the DVS image 920 through the dynamic vision sensor 102. TheRGB image 910 and the DVS image 920 may be transmitted to the AIprocessor 900. The AI processor 900 may input the RGB image 910 and theDVS image 920 to the second artificial intelligence model. In this case,the second artificial intelligence model may compare the RGB image 910with the DVS image 920 and determine that only a few features or edgesare detected from the tunnel exit region 911 in the RGB image 910 whilea lot of features or edges are detected from the tunnel exit region 921in the DVS image 920. In this case, the second artificial intelligencemodel may define the tunnel exit region 911 of the RGB image 910, as theregion of interest 930 where an object exists but is not recognized. Thesecond artificial intelligence model may communicate information aboutthe region of interest 930 to the AI processor 900.

In operation S860, the autonomous driving device 100 may control thephotographic configuration information of the camera 101 by using athird artificial intelligence model. In operation S870, the autonomousdriving device 100 may obtain the enhanced second RGB image based on thechanged photographic configuration information.

According to an embodiment, the third artificial intelligence model maybe a neural network model that learns from RGB images and thephotographic configuration information (e.g., exposure, white balance,focus) of the camera 101. The third artificial intelligence model may befor recommending appropriate photographic configuration information. Thethird artificial intelligence model may be separated from the firstartificial intelligence model and the second artificial intelligencemodel. Alternatively, the third artificial intelligence model may beintegrated with the first artificial intelligence model and the secondartificial intelligence model, to constitute a single model.

Referring to FIG. 10 , the autonomous driving device 100 may applyinformation about the RGB image 910 and the region of interest 930 tothe third artificial intelligence model. In this case, the thirdartificial intelligence model may determine a photographic configurationvalue for changing the brightness of the region of interest 930 of theRGB image 910. According to an embodiment, the third artificialintelligence model may modify at least one of exposure, focus, and whitebalance with respect to the region of interest 930. In particular, thethird artificial intelligence model may control an exposure value withrespect to the region of interest 930 by adjusting at least one of thegain, aperture, and exposure time of the camera 101. For example,because the region of interest 930 is a considerably bright region of atunnel exit, the third artificial intelligence model may determine aphotographic configuration value (e.g., a high gain value) to capturethe region of interest 930 appearing darker.

In a case where the autonomous driving device 100 changes the currentphotographic configuration values of the camera 101 to those determinedby the third artificial intelligence model, the camera 101 may obtain anenhanced RGB image 1000 based on the changed photographic configurationvalues. For example, the enhanced RGB image 1000 may be generally darkerthan the RGB image 910, and features or edges may appear in the regionof interest 1030 of the enhanced RGB image 1000. Therefore, theautonomous driving device 100 may recognize an object in the region ofinterest 1030 of the enhanced RGB image 1000.

Referring to FIG. 11 , the autonomous driving device 100 may obtain theRGB image 1110 through the camera 101 and the DVS image 1120 through thedynamic vision sensor 102 when entering a tunnel. The RGB image 1110 andthe DVS image 1120 may be transmitted to the AI processor 900. The AIprocessor 900 may input the RGB image 1110 and the DVS image 1120 to thesecond artificial intelligence model. In this case, the secondartificial intelligence model may compare the RGB image 1110 with theDVS image 1120 and determine that only a few features or edges aredetected from a tunnel entrance region 1111 in the RGB image 1110 whilea lot of features or edges are detected from the tunnel entrance region1121 in the DVS image 1120. In this case, the second artificialintelligence model may define the tunnel entrance region 1111 of the RGBimage 1110, as the region of interest in which an object exists but isnot recognized. The second artificial intelligence model may transmitinformation about the region of interest to the AI processor 900.

The AI processor 900 may apply the RGB image 1110 and the informationabout the region of interest to the third artificial intelligence model.In this case, the third artificial intelligence model may determine aphotographic configuration value for changing the brightness of theregion of interest (e.g., the tunnel entrance region 1111) of the RGBimage 1110. For example, because the region of interest is aconsiderably dark region of a tunnel entrance, the third artificialintelligence model may determine a photographic configuration value(e.g., a low gain value) to capture the region of interest appearingbrighter.

In a case where the autonomous driving device 100 changes the currentphotographic configuration values of the camera 101 to those determinedby the third artificial intelligence model, the camera 101 may obtain anenhanced RGB image 1130 based on the changed photographic configurationvalues. For example, the enhanced RGB image 1130 may be generallybrighter than the RGB image 1110, and features or edges may appear inthe region of interest 1131 of the enhanced RGB image 1130. Therefore,the autonomous driving device 100 may recognize an object in the regionof interest 1131 of the enhanced RGB image 1130.

Referring to FIG. 12 , the autonomous driving device 100 may obtain anRGB image 1210 through the camera 101 and a DVS image 1220 through thedynamic vision sensor 102, when a backlight occurs in the scene. The RGBimage 1210 and the DVS image 1220 may be transmitted to the AI processor900. The AI processor 900 may input the RGB image 1210 and the DVS image1220 to the second artificial intelligence model. In this case, thesecond artificial intelligence model may compare the RGB image 1210 withthe DVS image 1220 and determine that only a few features or edges aredetected from an upper left region 1211 in the RGB image 1210 while alot of features or edges are detected from the upper left region 1221 inthe DVS image 1220. In this case, the second artificial intelligencemodel may define the upper left region 1211 in the RGB image 1210, asthe region of interest in which an object exists but is not recognized.The second artificial intelligence model may transmit information aboutthe region of interest to the AI processor 900.

The AI processor 900 may apply the RGB image 1210 and the informationabout the region of interest to the third artificial intelligence model.In this case, the third artificial intelligence model may determine aphotographic configuration value for changing the brightness of theregion of interest (e.g., the upper left region 1211 appearing brightdue to the backlight) of the RGB image 1210. For example, because theregion of interest is a considerably bright region, the third artificialintelligence model may determine a photographic configuration value(e.g., a high gain value) to capture the region of interest appearingdarker. Alternatively, the third artificial intelligence model maycontrol exposure by adjusting the autofocus point or changing themetering mode.

In a case where the autonomous driving device 100 changes the currentphotographic configuration values of the camera 101 to those determinedby the third artificial intelligence model, the camera 101 may obtain anenhanced RGB image 1230 based on the changed photographic configurationvalues. For example, the enhanced RGB image 1230 may be generally darkerthan the RGB image 1210, and features or edges may appear in the regionof interest 1231 of the enhanced RGB image 1230. Therefore, theautonomous driving device 100 may recognize an object (e.g., a sign) inthe region of interest 1231 of the enhanced RGB image 1230.

In operation S880, in a case where no object has not been recognizedfrom the region of interest (a region(s) corresponding to the at leastone second region of the first RGB image) of the second RGB image, theautonomous driving device 100 may control the photographic configurationinformation of the camera 101 in relation to the at least one secondregion (region of interest) again. For example, in a case of the regionof interest being a dark region, the autonomous driving device 100 mayfurther increase the gain of the camera 101 to capture the region ofinterest appearing brighter.

In operation S890, the autonomous driving device 100 may obtain theposition information of the autonomous driving device 100 by using arecognized object.

Operation S890 corresponds to operation S590 of FIG. 5 , its detaileddescription will be omitted.

Hereinafter, an operation of, in a case where the autonomous drivingdevice 100 has defined a plurality of regions of interest, controllingthe photographic configuration information of the camera 101 accordingto priorities of the plurality of regions of interest will be describedin detail with reference to FIGS. 13 and 14 .

FIG. 13 is a flowchart for explaining a method of controllingphotographic configuration information of a camera according topriorities of a plurality of regions of interest, according to anembodiment.

In operation S1310, the autonomous driving device 100 may obtain thefirst RGB image by using the camera 101.

Operation S1310 corresponds to operation S210 of FIG. 2 , its detaileddescription will be omitted.

In operation S1320, the autonomous driving device 100 may predict aplurality of object-unrecognizable regions in the first RGB image basedon the brightness information of the first RGB image.

According to an embodiment, the autonomous driving device 100 maypredict the plurality of object-unrecognizable regions in the first RGBimage by using the histogram of the first RGB image or the artificialintelligence model.

For example, the autonomous driving device 100 may define, as theobject-unrecognizable region, a region in which the brightness valuesare out of the threshold range in the first RGB image (e.g., aconsiderably dark or bright region). In this case, in a case where aplurality of regions in which the brightness values are out of thethreshold range exist in the first RGB image, the autonomous drivingdevice 100 may detect a plurality of object-unrecognizable regions.

Operation S1320 corresponds to operation S220 of FIG. 2 , its detaileddescription will be omitted.

In operation S1330, the autonomous driving device 100 may determine theplurality of regions of interest in which an object exists, from amongthe plurality of object-unrecognizable regions, based on the objectinformation obtained through the dynamic vision sensor.

For example, the autonomous driving device 100 may define a plurality ofregions in which an object is not captured by the camera 101 but theobject is detected by the dynamic vision sensor 102, as the plurality ofregions of interest. In this case, brightness values of the plurality ofregions of interest may be different from each other. For example, afirst region of interest may be a dark region while a second region ofinterest may be a bright region. Accordingly, it may be difficult todetect all objects in each of the plurality of regions of interest byadjusting the photographic configuration information of the camera 101only once.

Therefore, in operation S1340, the autonomous driving device 100 maydetermine priorities of the plurality of regions of interest accordingto a predefined criterion.

According to an embodiment, the autonomous driving device 100 may assigna region having a low brightness a higher priority. For example, a darkregion may have a higher priority than that of a bright region.Alternatively, the autonomous driving device 100 may assign a regionhaving a higher brightness a higher priority. For example, a brightregion may have a higher priority than that of a dark region.

According to an embodiment, the autonomous driving device 100 maydetermine the priorities based on areas of the regions of interest. Forexample, a priority of a wide region may be higher than that of a narrowregion.

Meanwhile, according to an embodiment, the autonomous driving device 100may determine the priorities of the regions of interest by usingsurrounding environment information (e.g., context information). Forexample, in a case where the autonomous driving device 100 is at thetunnel entrance, a region having a low brightness may be assigned a highpriority, whereas, in a case where the autonomous driving device 100 isat the tunnel exit, a region having high brightness may be assigned ahigh priority, but the present disclosure is not limited thereto.

In operations S1350 to S1380, the autonomous driving device 100 mayobtain an enhanced RGB image by controlling the photographicconfiguration information of the camera 101, considering the prioritiesof the plurality of regions of interest. That is, in operation S1350,the autonomous driving device 100 may obtain an enhanced (n+1)th RGBimage by controlling the photographic configuration information of thecamera 101 in relation to the region of interest having an n-thpriority. In operation S1360, the autonomous driving device 100 mayrecognize an n-th object in the (n+1)th RGB image. In operation S1370,the autonomous driving device 100 may determine whether a region ofinterest having an n-th priority is the last region of interest. Inoperation S1380, in a case where the region of interest having the n-thpriority is not the last region of interest, the autonomous drivingdevice 100 may select the region of interest having the (n+1)th priorityand perform operations from S1350 again.

For example, the autonomous driving device 100 may obtain the enhancedsecond RGB image by controlling the photographic configurationinformation of the camera 101 in relation to the region of interesthaving a first priority. In this case, the autonomous driving device 100may recognize a first object in the region of interest (regioncorresponding to the region of interest having the first priority) ofthe second RGB image. Then, the autonomous driving device 100 may obtainan enhanced third RGB image by controlling the photographicconfiguration information of the camera 101 in relation to a region ofinterest having a second priority. The autonomous driving device 100 mayrecognize a second object in a region of interest (region correspondingto the region of interest having the second priority) of the third RGBimage. Here, the first object and the second object may be differentfrom each other. The autonomous driving device 100 may determine whetherthe region of interest having the second priority is the last region ofinterest. When the region of interest having the second priority is notthe last region of interest, the autonomous driving device 100 mayobtain an enhanced fourth RGB image by controlling the photographicconfiguration information of the camera 101 in relation to a region ofinterest having a third priority, and recognize a third object in theregion of interest (region corresponding to the region of interesthaving the third priority) of the enhanced fourth RGB image. That is,the autonomous driving device 100 may control the photographicconfiguration information of the camera 101 such that objects may besequentially detected from the regions of interest according to theirpriorities.

Referring to FIG. 14 , an operation of, by the autonomous driving device100, controlling the photographic configuration information of thecamera 101, considering the priorities of the plurality of regions ofinterest will be described in detail.

FIG. 14 is a diagram for explaining priorities of a plurality of regionsof interest, according to an embodiment.

The autonomous driving device 100 may obtain an RGB image 1410 throughthe camera 101 while passing through a tunnel, and obtain a DVS image1420 through the dynamic vision sensor 102. In this case, the autonomousdriving device 100 may compare the RGB image 1410 with the DVS image1420 and determine that only a few features or edges are detected from alower left region (region {circle around (1)}) in the RGB image 1410,but a lot of features or edges are detected from the lower left region(region {circle around (1)}) in the DVS image 1420. In addition, theautonomous driving device 100 may determine that only a few features oredges are detected from a tunnel exit region (region {circle around(2)}) in the RGB image 1410, but a lot of features or edges are detectedfrom the tunnel exit region (region {circle around (2)}) in the DVSimage 1420. In this case, the autonomous driving device 100 may definethe lower left region (region {circle around (1)}) and the tunnel exitregion (region {circle around (2)}) in the RGB image 1410, as theregions of interest.

In a case where a region of interest having a low brightness is set tobe assigned a high priority, the autonomous driving device 100 mayassign the lower left region (region {circle around (1)}) a higherpriority than that of the tunnel exit region (region {circle around(2)}).

In this case, the autonomous driving device 100 may control thephotographic configuration information of the camera 101 based on thelower left region (region {circle around (1)}) first. For example,because the lower left region (region {circle around (1)}) is a darkregion, the autonomous driving device 100 may increase the gain tocapture the lower left region (region {circle around (1)}) appearingbrighter. Here, the lower left region (region {circle around (1)})appears brighter, the autonomous driving device 100 may recognize anobject (e.g., an external vehicle) in the lower left region (region{circle around (1)}). Because the gain has been increased andaccordingly, the tunnel exit region (region {circle around (2)}) appearseven brighter, an object may still not be detected in the tunnel exitregion (region {circle around (2)}).

Then, the autonomous driving device 100 may control the photographicconfiguration information of the camera 101 based on the tunnel exitregion (region {circle around (2)}). For example, because the tunnelexit region (region {circle around (2)}) is a bright area, theautonomous driving device 100 may decrease the gain to capture thetunnel exit region (region {circle around (2)}) appearing darker. Here,the tunnel exit region (region {circle around (2)}) appears darker, theautonomous driving device 100 may recognize an object (e.g., a lane,pedestrian bridge, street tree) in the tunnel exit region (region{circle around (2)}).

FIG. 15 is a flowchart for explaining a method of, by an autonomousdriving device, tracking an object, according to an embodiment.

In operation S1510, the autonomous driving device 100 may select anoperation mode for object detection. Operation modes for object trackingmay include, but is not limited to, a high-speed detection mode and anentire region detection mode.

The high-speed detection mode refers to a mode for detecting an objectat a high speed, by performing image processing on a candidate regionhaving a high possibility of detecting an object in an RGB image. Thecandidate region may be determined based on information detected by thedynamic vision sensor 102.

The entire region detection mode refers to a mode for detecting anobject by performing image processing on an entire region of an RGBimage.

In operations S1520 and S1530, when the high-speed detection mode isselected, the autonomous driving device 100 may operate in thehigh-speed detection mode.

According to an embodiment, in a case where a new object has not beendetected by the dynamic vision sensor 102, the autonomous driving device100 may not perform a new object recognition process on the RGB image.

In operation S1540, the autonomous driving device 100 may detect a newobject appearing around the autonomous driving device 100 through thedynamic vision sensor. Here, the new object may include a dynamic object(e.g., a vehicle, motorcycle, pedestrian).

Because the dynamic vision sensor 102 obtains data on a per-pixel basisrather than a frame basis, the new object may be detected earlier thanthe camera 101 does.

According to an embodiment, the new object appearing around theautonomous driving device 100 may be detected by comparing the currentframe with the previous frame of the DVS image. For example, when anexternal vehicle appears in a second lane at the right of the first lanein which the autonomous driving device 100 is driving, the dynamicvision sensor 102 may detect the external vehicle earlier than thecamera 101 does. Here, an outline of the external vehicle may appear ina right region of the DVS image.

In operation S1550, the autonomous driving device 100 may determine thecandidate region in which the possibility of recognizing a new object inthe RGB image is greater than a threshold value.

According to an embodiment, the autonomous driving device 100 maydetermine the candidate region in the RGB image based on informationabout from where the new object appears on the DVS image. For example,in a case where the new object appears from a right region of theautonomous driving device 100 as a result of analyzing the DVS image,the autonomous driving device 100 may define the right region in the RGBimage as the candidate region.

In operation S1560, the autonomous driving device 100 may recognize thenew object from the candidate region of the RGB image by performingimage processing on the candidate region.

According to an embodiment, the autonomous driving device 100 mayextract at least one feature that constitutes the object, from thecandidate region. The autonomous driving device 100 may recognize theobject in the candidate region by using the extracted at least onefeature. For example, the autonomous driving device 100 may recognizethe external vehicle that is traveling in the right lane in thecandidate region of the RGB image. In this case, because the autonomousdriving device 100 does not need to perform the image processing on theentire region of the RGB image in order to recognize the new object, aspeed and accuracy in recognizing an object may be improved.

In operation S1570, when the high-speed detection mode is not selected,the autonomous driving device 100 may operate in the entire regiondetection mode.

According to an embodiment, in a case where an image processingcapability is sufficient or in a case of a critical event, theautonomous driving device 100 may select the entire region detectionmode. Alternatively, the autonomous driving device 100 may periodicallyoperate in the entire region detection mode.

In operation S1590, when the autonomous driving device 100 operates inthe entire region detection mode, the autonomous driving device 100 mayrecognize the new object by performing the image processing on theentire region of the RGB image.

In operation S1590, the autonomous driving device 100 may track the newobject by using the camera 101.

According to an embodiment, the autonomous driving device 100 may tracka change in the new object based on feature information of the newobject extracted from a series of frames of an RGB image. For example,the autonomous driving device 100 may track a change in a position ofthe new object. According to an embodiment, the autonomous drivingdevice 100 may mark an identification image around an object that isbeing tracked.

According to an embodiment, some of operations S1510 to S1590 may beomitted, and the order of some of operations S1510 to S1590 may bechanged.

FIG. 16 is a diagram for explaining an operation of, by an autonomousdriving device, recognizing and tracking a new object detected by adynamic vision sensor using a camera.

Referring to a first RGB image 1610 of FIG. 16 , the autonomous drivingdevice 100 may recognize and track objects moving in front of theautonomous driving device 100 by using a front camera 101. For example,the autonomous driving device 100 may recognize and track a firstvehicle 1601, a second vehicle 1602, and a third vehicle 1603.

Referring to a DVS image 1620 of FIG. 16 , the autonomous driving device100 may detect a new object 1621 approaching the autonomous drivingdevice 100 by using the dynamic vision sensor 102. For example, theautonomous driving device 100 may detect an outline of a fourth vehicle1621 approaching the left of the autonomous driving device 100, from theDVS image 1620.

Referring to a second RGB image 1630 of FIG. 16 , in a case where thenew object has been detected through the dynamic vision sensor 1620, theautonomous driving device 100 may determine the candidate region inwhich the probability of recognizing the new object in the second RGBimage 1630 obtained through the camera 101 is greater than the thresholdvalue. For example, in a case where the autonomous driving device 100has recognized that the new object is approaching the left of theautonomous driving device 100, through the DVS image 1620, and thus theautonomous driving device 100 may define a left region 1631 of thesecond RGB image 1630 as the candidate region.

The autonomous driving device 100 may recognize the fourth vehicle 1621by performing the image processing on the left region 1631 of the secondRGB image 1630. In addition, the autonomous driving device 100 may trackthe fourth vehicle 1621 together with the first to third vehicles 1601to 1603 by using the camera 101.

According to an embodiment, the autonomous driving device 100 maypredict a presence and position of the new object through the dynamicvision sensor 102, and thus may rapidly recognize and track the newobject on the RGB image.

FIG. 17 is a block diagram for explaining a configuration of anautonomous driving device, according to an embodiment.

Referring to FIG. 17 , the autonomous driving device 100 may include asensing unit 110, a processor 120, a communicator 130, a driving unit140, an outputter 150, a storage unit 160, and an inputter 170. However,all components shown in FIG. 17 are not indispensable components of theautonomous driving device 100. The autonomous driving device 100 may beimplemented by more components than the components illustrated in FIG.17 , or by fewer components than the components illustrated in FIG. 17 .For example, as shown in FIG. 1 , the autonomous driving device 100 mayinclude the camera 101, the dynamic vision sensor 102, and the processor120. The components will be described in order.

The sensing unit 110 may include a plurality of sensors configured todetect information about a surrounding environment of the autonomousdriving device 100. For example, the sensing unit 110 may include thecamera 101 (e.g., a stereo camera, a mono camera, a wide angle camera,an around-view camera, or a three-dimensional vision sensor), thedynamic vision sensor 102, a lidar sensor 103, a radar sensor 104, aninertial sensor (inertial measurement unit (IMU)) 105, an ultrasonicsensor 106, an infrared sensor 107, a distance sensor 108, atemperature/humidity sensor 109, a position sensor 111 (e.g., a globalpositioning system (GPS), differential GPS (DGPS), an inertialnavigation system (INS)), and a motion sensing unit 112, but is notlimited thereto.

The motion sensing unit 112 may detect a motion of the autonomousdriving device 100, and may include, for example, a geomagnetic sensor113, an acceleration sensor 114, and a gyroscope sensor 115, but is notlimited thereto.

According to an embodiment, the camera 101 may include a plurality ofcameras, and the plurality of cameras may be arranged at a plurality oflocations inside or outside the autonomous driving device 100. Forexample, three cameras may be arranged in a front portion, one cameramay be arranged in a rear portion, two cameras may be arranged in a leftside portion, and two cameras may be arranged in a right side portion ofthe autonomous driving device 100, but the present disclosure is notlimited thereto. A function of each sensor may be intuitively deducedfrom the name by one of ordinary skill in the art, and thus a detaileddescription thereof is omitted.

The processor 120 may generally control the overall operation of theautonomous driving device 100. The processor 120 may control the sensingunit 110, the communicator 130, the driving unit 140, the outputter 150,the storage unit 160, and the inputter 170 by executing programs storedin the storage unit 160.

The processor 120 may obtain the first RGB image by using the camera101.

The processor 120 may analyze the first RGB image to obtain thehistogram of the first RGB image, and determine whether theobject-unrecognizable region exists in the first RGB image by using thehistogram of the first RGB image.

In a case where it has been determined that the object-unrecognizableregion exists in the first RGB image, the processor 120 may predict theat least one first region in the first RGB image based on the brightnessinformation of the first RGB image. For example, the processor 120 maydefine, as the at least one first region, a region in which thebrightness values are out of the threshold range in the first RGB image.

The processor 120 may determine the at least one second region in whichan object exists from among the at least one first region, based on theobject information obtained through the dynamic vision sensor 102. Theprocessor 120 may obtain the enhanced second RGB image by controllingthe photographic configuration information of the camera 101 in relationto the at least one second region. For example, the processor 120 maycontrol the photographic configuration information of the camera 101 byadjusting at least one of the gain, aperture, and exposure time of thecamera 101.

The processor 120 may recognize the object in the second RGB image. Theprocessor 120 may track the object recognized in the second RGB image byusing the camera 120. The processor 120 may detect the new objectappearing around the autonomous driving device 100 through the dynamicvision sensor 102, and may determine the candidate region in which theprobability of recognizing the new object in the third RGB imageobtained through the camera 101 is greater than the threshold value. Theprocessor 120 may recognize the new object detected by the dynamicvision sensor 102 from the third RGB image by performing the imageprocessing on the candidate region.

The processor 120 may set the frame rate of the dynamic vision sensor102 to be the same as that of the camera 101.

According to an embodiment, the processor 120 may include the artificialintelligence (AI) processor. In this case, the AI processor maydetermine whether the object-unrecognizable region exists in the firstRGB image by using the first artificial intelligence model that has beentrained from a plurality of RGB images, and when it has been determinedthat the object-unrecognizable region exists in the first RGB image, theAI processor may predict the at least one first region(object-unrecognizable region) in the first RGB image by using the firstartificial intelligence model. In addition, the processor 120 may plan amotion of the autonomous driving device 100 by using a learned networkmodel of an AI system.

The AI processor may be manufactured in the form of an AI-dedicatedhardware chip or may be manufactured as part of an existing generalpurpose processor (e.g., a CPU or application processor) or a dedicatedgraphics processor (e.g., a GPU) and mounted on the autonomous drivingdevice 100.

The communicator 130 may include at least one antenna for wirelesslycommunicating with another device (e.g., an external vehicle or externalserver). For example, the communicator 130 may include one or morecomponents that allow communication between the autonomous drivingdevice 100 and an external vehicle or between the autonomous drivingdevice 100 and a server. For example, the communicator 130 may include ashort-range wireless communicator 131, a mobile communicator 132, and abroadcast receiver 133, but is not limited thereto.

The short-range wireless communicator 131 may include, but is notlimited to, a Bluetooth communicator, a Bluetooth low energy (BLE)communicator, a near field communicator (NFC), a Wi-Fi (WLAN)communicator, a Zigbee communicator, an infrared data association (IrDA)communicator, a Wi-Fi direct (WFD) communicator, an ultra wideband (UWB)communicator, an Ant+ communicator, a microwave communicator, etc.

The mobile communicator 132 may transmit and receive a wireless signalto and from at least one of a base station, an external terminal, and aserver on a mobile communication network. Here, the wireless signals mayinclude various types of data based on transmission and reception ofvoice call signals, video call signals, or text/multimedia messages.

The broadcast receiver 133 may receive broadcast signals and/orbroadcast-related information through broadcast channels from outside.The broadcast channels may include satellite channels and terrestrialchannels. According to an embodiment, the autonomous driving device 100may not include the broadcast receiver 133.

According to embodiments, the communicator 130 may performvehicle-to-vehicle (V2V) communication with an external vehicle locatedwithin a certain distance from the autonomous driving device 100, orperform vehicle-to-infrastructure (V2I) communication withinfrastructure located within a certain distance from the autonomousdriving device 100. For example, the communicator 130 may broadcast oradvertise a packet including identification information, a position, orspeed of the autonomous driving device 100. Also, the communicator 130may receive a packet broadcasted or advertised by the external vehicle.

The driving unit 140 may include elements used for driving (operating)the autonomous driving device 100 and for performing operations ofdevices in the autonomous driving device 100. The driving unit 140 mayinclude at least one of a power supply 141, a propelling unit 142, atraveling unit 143, and a peripheral device unit 144, but is not limitedthereto.

The peripheral device unit 144 may include a navigation system, a light,a turn signal lamp, a wiper, an internal light, a heater, and an airconditioner. The navigation system may be a system configured todetermine a driving route for the autonomous driving device 100. Thenavigation system may be configured to dynamically update the drivingroute while the autonomous driving device 100 is traveling. For example,the navigation system may utilize data collected by a GPS module todetermine the driving route for the autonomous driving device 100.

The outputter 150 may output an audio signal, a video signal, or avibration signal, and may include a display 151, an audio outputter 152,a vibration unit 153, etc.

The display 151 may display and output information processed in theautonomous driving device 100. For example, the display 151 may displaya map including a driving route, display positions of external vehicles,display blind spots of drivers of the external vehicles, or display acurrent speed, a remaining fuel amount, information for guiding thedriving route of the autonomous driving device 100, etc., but is notlimited thereto. The display 151 may display a user interface (UI) or agraphic user interface (GUI) associated with a call in a call mode.

Meanwhile, when the display 151 and a touch pad have a layer structureand are configured as a touch screen, the display 151 may be used as aninput device in addition to an output device. The display 151 mayinclude at least one of a liquid crystal display, a thin filmtransistor-liquid crystal display, an organic light-emitting diode, aflexible display, a three-dimensional (3D) display, an electrophoreticdisplay, etc. The autonomous driving device 100 may include two or moredisplays 151 according to an implementation of the device 100.

According to one embodiment, the display 151 may include a transparentdisplay. The transparent display may be implemented in a projection typein addition to a transparent liquid crystal display (LCD) type, atransparent thin-film electroluminescent panel (TFEL) type, and atransparent organic light emitting diode (OLED) type. The projectiontype refers to a method of projecting and displaying an image on atransparent screen such as a head-up display (HUD).

The sound outputter 152 may output audio data received from thecommunicator 130 or stored in the storage unit 160. In addition, thesound outputter 152 may output a sound signal related to a functionperformed in the autonomous driving device 100. For example, the soundoutputter 152 may output a voice message for guiding the driving routeof the autonomous driving device 100. The audio outputter 152 mayinclude a speaker, a buzzer, etc.

The vibration unit 153 may output a vibration signal. For example, thevibration unit 153 may output a vibration signal corresponding to anoutput of audio data or video data (e.g., a warning message).

The storage unit 160 may store a program for processing and control ofthe processor 120, and may store input/output data (e.g., an RGB image,DVS image, road situation information, precision map, histogram). Thestorage unit 160 may store an artificial intelligence model 161.

The storage unit 160 may include at least one type of storage medium ofa flash memory type, a hard disk type, a multimedia card micro type, acard type memory (for example, SD or XD memory), random access memory(RAM), a static random access memory (SRAM), read only memory (ROM),electrically erasable programmable read-only memory (EEPROM),programmable read-only memory (PROM), a magnetic memory, a magneticdisk, an optical disk, or the like. Also, the autonomous driving device100 may operate a web storage or a cloud server that performs a storagefunction on the Internet.

The inputter 170 refers to a means through which a user inputs data forcontrolling the autonomous driving device 100. For example, the inputter170 may include a key pad, a dome switch, a touch pad (contact typecapacitance type, pressure type resistive type, infrared ray detectiontype, surface ultrasonic wave conduction type, an integral tensionmeasurement type, a piezo effect type, etc.), a jog wheel, a jog switch,and the like, but is not limited thereto.

FIG. 18 is a block diagram of a processor, according to an embodiment.

Referring to FIG. 18 , the processor 120 may include a data learner 1310and a data recognizer 1320.

The data learner 1310 may learn a standard for determining an objectrecognition situation. For example, the data learner 1310 may learn astandard for determining a situation in which it is difficult torecognize an object through the camera 101 (e.g., entering a tunnel,exiting a tunnel, backlighted in evening or dawn, driving at night,passing through a region with extreme changes in illumination, passingthrough a shadowed region). Also, the data learner 1310 may learn astandard for identifying the object-unrecognizable region in the RGBimage, or a standard for determining the region of interest in the RGBimage, based on the object information of the dynamic vision sensor 102.The data learner 1310 may also learn a standard about which data is usedto determine the photographic configuration information of the camera101 and how to determine the photographic configuration information byusing the data. The data learner 1310 may obtain data (e.g., an image)to be used for learning, apply the obtained data to a data recognitionmodel that will be described below, and learn a standard for recognizingan object through the camera 101

According to an embodiment, the data learner 1310 may learn personalizeddata. For example, the data learner 1310 may learn RGB images, contextinformation, etc. obtained from a route through which the autonomousdriving device 100 frequently travels. According to an embodiment, thedata learner 1310 may learn a standard for planning a motion of theautonomous driving device 100, or may learn a standard for recognizing aposition of the autonomous driving device 100.

The data recognizer 1320 may determine the object recognition situationbased on the data. The data recognizer 1320 may determine the objectrecognition situation from the detected data by using the trained datarecognition model. The data recognizer 1320 may obtain image data (e.g.,an RGB image or DVS image) according to a predefined standard bylearning, and use the data recognition model by using the obtained imagedata as an input value to perform object recognition based on the imagedata. In addition, a result value output by the data recognition modelby using the obtained image data as the input value may be used torefine the data recognition model.

At least one of the data learner 1310 and the data recognizer 1320 maybe manufactured in the form of at least one hardware chip and mounted onthe autonomous driving device 100. For example, at least one of the datalearner 1310 and the data recognizer 1320 may be manufactured in theform of a dedicated hardware chip for artificial intelligence (AI), ormay be manufactured as a part of an existing general purpose processor(e.g., a CPU or application processor) or a dedicated graphics processor(e.g., a GPU) and mounted on the autonomous driving device 100.

In this case, the data learner 1310 and the data recognizer 1320 may bemounted on a single autonomous driving device 100, or may be separatelymounted on electronic devices. For example, one of the data learner 1310and the data recognizer 1320 may be included in the autonomous drivingdevice 100, and the remaining one may be included in a server 200. Also,model information established by the data learner 1310 may be providedto the data recognizer 1320 and data input to the data recognizer 1320may be provided as additional training data to the data learner 1310 bywire or wirelessly.

At least one of the data learner 1310 and the data recognizer 1320 maybe implemented as a software module. When at least one of the datalearner 1310 and the data recognizer 1320 is implemented as a softwaremodule (or a program module including instructions), the software modulemay be stored in a non-transitory computer-readable recording medium.Also, in this case, at least one software module may be provided by anoperating system (OS) or a predefined application. Alternatively, a partof at least one software module may be provided by an operating system(OS), and the remaining part may be provided by a predefinedapplication.

FIG. 19 is a block diagram of the data learner 1310, according to anembodiment.

Referring to FIG. 19 , the data learner 1310 according to an embodimentmay include a data obtainer 1310-1, a preprocessor 1310-2, a trainingdata selector 1310-3, a model learner 1310-4, and a model evaluator1310-5.

The data obtainer 1310-1 may obtain data needed to determine the objectrecognition situation. The data obtainer 1310-1 may obtain data (e.g.,an RGB image or DVS image) necessary for learning to determine theobject recognition situation. According to an embodiment, the dataobtainer 1310-1 may directly generate data needed to determine theobject recognition situation or may receive the data needed to determinethe object recognition situation from an external device or a server.

According to an embodiment, the data needed to determine the objectrecognition situation may include, but is not limited to, an RGB image,object information of the dynamic vision sensor 100, surroundingenvironment information of the autonomous driving device 100,personalized training data, etc.

The preprocessor 1310-2 may preprocess the obtained data to be used forlearning to determine the object recognition situation. Thepre-processor 1310-2 may process the obtained data into a predefinedformat such that the model learner 1310-4 that will be described belowmay use the obtained data for learning to determine the objectrecognition situation.

The training data selector 1310-3 may select data needed for learningfrom among the pieces of preprocessed data. The selected data may beprovided to the model learner 1310-4. The training data selector 1310-3may select the data needed for learning from the preprocessed dataaccording to a predefined standard for determining the objectrecognition situation. Also, the training data selector 1310-3 mayselect data based on a predefined standard according to learning by themodel learner 1310-4 that will be described below.

The model learner 1310-4 may learn a standard about how to determine theobject recognition situation based on the training data. In addition,the model learner 1310-4 may learn a standard about which training datais to be used to determine the object recognition situation.

In addition, the model learner 1310-4 may train a data recognition modelused to determine the object recognition situation by using the trainingdata. In this case, the data recognition model may be a model that ispre-established. For example, the data recognition model may be a modelthat is pre-established by receiving basic training data (e.g., sampleimages).

The data recognition model may be established in consideration of afield to which a recognition model is applied, the purpose of learning,or the computer performance of the autonomous driving device 100. Thedata recognition model may be, for example, a model based on a neuralnetwork. For example, a model such as a deep neural network (DNN), arecurrent neural network (RNN), or a bidirectional recurrent deep neuralnetwork (BRDNN) may be used as the data recognition model, but thepresent disclosure is not limited thereto.

According to various embodiments, when a plurality of data recognitionmodels that are pre-established exist, the model learner 1310-4 maydetermine a data recognition model having a high relationship betweeninput training data and basic training data as the data recognitionmodel to be trained. In this case, the basic training data may bepre-classified according to types of data, and the data recognitionmodel may be pre-established according to the types of data. Forexample, the basic training data may be pre-classified according tovarious standards such as an area where the training data is generated,a time for which the training data is generated, a size of the trainingdata, a genre of the training data, a generator of the training data,and a type of the subject in the training data.

Also, the model learner 1310-4 may train the data recognition model byusing a learning algorithm including, for example, errorback-propagation or gradient descent.

Also, the model learner 1310-4 may train the data recognition modelthrough supervised learning by using, for example, the training data asan input value. Also, the model learner 1310-4 may train the datarecognition model through unsupervised learning to find a standard fordetermining a situation by learning a type of data needed to determinethe situation by itself without supervision. Also, the model learner1310-4 may train the data recognition model through reinforcementlearning using a feedback about whether a result of determining theobject recognition situation according to learning is right.

Also, when the data recognition model has been trained, the modellearner 1310-4 may store the trained data recognition model. In thiscase, the model learner 1310-4 may store the trained data recognitionmodel in the storage unit 160 of the autonomous driving device 100including the data recognizer 1320. Alternatively, the model learner1310-4 may store the trained data recognition model in the storage unit160 of the autonomous driving device 100 including the data recognizer1320 that will be described below. Alternatively, the model learner1310-4 may store the trained data recognition model in a memory of theserver 200 connected to the autonomous driving device 100 through awired or wireless network.

In this case, the storage unit 160 in which the trained data recognitionmodel is stored may also store, for example, a command or data relatedto at least one other component of the autonomous driving device 100.Also, the storage unit 160 may store software and/or programs. Theprograms may include, for example, a kernel, middleware, an applicationprogramming interface (API) and/or an application program (or“application”).

The model evaluator 1310-5 may input evaluation data to the datarecognition model, and may allow the model learner 1310-4 to re-trainthe data recognition model when a recognition result output on theevaluation data does not satisfy a predefined criterion. In this case,the evaluation data may be predefined data for evaluating the datarecognition model.

For example, from among recognition results of the trained datarecognition model output on the evaluation data, when the number or aratio of incorrect recognition results exceeds a predefined thresholdvalue, the model evaluator 1310-5 may evaluate that the predefinedcriterion is not satisfied. For example, when the predefined criterionis 2% and incorrect recognition results are output on more than 20pieces of evaluation data from among 1000 pieces of evaluation data, themodel evaluator 1310-5 may evaluate that the trained data recognitionmodel is not suitable.

When a plurality of trained data recognition models exist, the modelevaluator 1310-5 may evaluate whether each of the trained datarecognition models satisfies the predefined criterion, and may define amodel satisfying the predefined criterion as a final data recognitionmodel. In this case, when a plurality of models satisfy the predefinedcriterion, the model evaluator 1310-5 may define one that is preset or apreset number of models in a descending order of evaluation scores asfinal data recognition models.

At least one of the data obtainer 1310-1, the pre-processor 1310-2, thetraining data selector 1310-3, the model learner 1310-4, and the modelevaluator 1310-5 in the data learner 1310 may be manufactured as atleast one hardware chip and may be mounted on the autonomous drivingdevice 100. For example, at least one of the model learner 1310-4, thepre-processor 1310-2, the training data selector 1310-3, the modellearner 1310-4, and the model evaluator 1310-5 may be manufactured inthe form of a dedicated hardware chip for artificial intelligence (AI),or may be manufactured as a part of an existing general purposeprocessor (e.g., a CPU or application processor) or a dedicated graphicsprocessor (e.g., a GPU) and mounted on the autonomous driving device100.

Also, the data obtainer 1310-1, the preprocessor 1310-2, the trainingdata selector 1310-3, the model learner 1310-4, and the model evaluator1310-5 may be mounted on a single autonomous driving device 100, or maybe separately mounted on electronic devices For example, some of thedata obtainer 1310-1, the preprocessor 1310-2, the training dataselector 1310-3, the model learner 1310-4, and the model evaluator1310-5 may be included in the autonomous driving device 100, and therest may be included in the server 200.

At least one of the data obtainer 1310-1, the preprocessor 1310-2, thetraining data selector 1310-3, the model learner 1310-4, and the modelevaluator 1310-5 may be implemented as a software module. When at leastone of the data obtainer 1310-1, the preprocessor 1310-2, the trainingdata selector 1310-3, the model learner 1310-4, and the model evaluator1310-5 is implemented as a software module (or a program moduleincluding instructions), the software module may be stored in anon-transitory computer readable medium. In this case, the at least onesoftware module may be provided by an operating system (OS) or apredefined application. Alternatively, a part of the at least onesoftware module may be provided by an operating system (OS), and theremaining part may be provided by a predefined application.

FIG. 20 is a block diagram of the data recognizer 1320, according to anembodiment.

Referring to FIG. 20 , the data recognizer 1320 according to anembodiment may include a data obtainer 1320-1, a preprocessor 1320-2, arecognition data selector 1320-3, a recognition result provider 1320-4,and a model refiner 1320-5.

The data obtainer 1320-1 may obtain the data needed to determine theobject recognition situation, and the preprocessor 1320-2 may preprocessthe obtained data such that the obtained data may be used to determinethe object recognition situation. The preprocessor 1320-2 may processthe obtained data into a predefined format such that the recognitionresult provider 1320-4 that will be described below may use the obtaineddata for determining the object recognition situation.

The recognition data selector 1320-3 may select the data needed todetermine the object recognition situation from among the pieces ofpreprocessed data. The selected data may be provided to the recognitionresult provider 1320-4. The recognition data selector 1320-3 may selectsome or all of the pieces of preprocessed data according to a presetstandard for determining the object recognition situation. Also, therecognition data selector 1320-3 may select data according to a standardpreset by learning by the model learner 1310-4 as described below.

The recognition result provider 1320-4 may determine the objectrecognition situation by applying the selected data to the datarecognition model. The recognition result provider 1320-4 may provide arecognition result according to recognition purpose of the data. Therecognition result provider 1320-4 may apply the selected data to thedata recognition model by using the data selected by the recognitiondata selector 1320-3 as an input value. Also, the recognition result maybe determined by the data recognition model.

For example, a recognition result of at least one image may be providedas text, a voice, a video, an image, or instructions (e.g., applicationexecution instructions or module function execution instructions). Forexample, the recognition result provider 1320-4 may provide arecognition result of an object included in the at least one image. Therecognition result may include, for example, pose information of theobject included in the at least one image, surrounding state informationof the object, and motion change information of the object included in avideo.

The model refiner 1320-5 may refine the data recognition model based onevaluation of the recognition result provided by the recognition resultprovider 1320-4. For example, the model refiner 1320-5 may provide therecognition result provided by the recognition result provider 1320-4 tothe model learner 1310-4 such that the model learner 1340-4 refines thedata recognition model.

At least one of the data obtainer 1320-1, the preprocessor 1320-2, therecognition data selector 1320-3, the recognition result provider1320-4, and the model refiner 1320-5 in the data recognizer 1320 may bemanufactured as at least one hardware chip and may be mounted on theautonomous driving device 100. For example, at least one of the dataobtainer 1320-1, the preprocessor 1320-2, the recognition data selector1320-3, the recognition result provider 1320-4, and the model refiner1320-5 may be manufactured in the form of a dedicated hardware chip forartificial intelligence (AI), or may be manufactured as a part of anexisting general purpose processor (e.g., a CPU or applicationprocessor) or a dedicated graphics processor (e.g., a GPU) and mountedon the autonomous driving device 100.

Also, the data obtainer 1320-1, the preprocessor 1320-2, the recognitiondata selector 1320-3, the recognition result provision unit 1320-4, andthe model refiner 1320-5 may be mounted on a single autonomous drivingdevice 100, or may be separately mounted on electronic devices. Forexample, some of the data obtainer 1320-1, the preprocessor 1320-2, therecognition data selector 1320-3, the recognition result provision unit1320-4, and the model refiner 1320-5 may be included in the autonomousdriving device 100, and the rest may be included in a server 200.

At least one of the data obtainer 1320-1, the preprocessor 1320-2, therecognition data selector 1320-3, the recognition result provider1320-4, and the model refiner 1320-5 may be implemented as a softwaremodule. When at least one of the data obtainer 1320-1, the preprocessor1320-2, the recognition data selector 1320-3, the recognition resultprovider 1320-4, and the model refiner 1320-5 is implemented as asoftware module (or a program module including instructions), thesoftware module may be stored in a non-transitory computer readablemedium. Also, in this case, at least one software module may be providedby an operating system (OS) or a predefined application. Alternatively,a part of at least one software module may be provided by an operatingsystem (OS), and the remaining part may be provided by a predefinedapplication.

FIG. 21 is a diagram illustrating an example in which the autonomousdriving device 100 and the server 200 interoperate to learn andrecognize data, according to an embodiment.

Referring to FIG. 21 , the server 200 may learn a standard fordetermining the object recognition situation, and the autonomous drivingdevice 100 may determine the object recognition situation based on aresult of learning by the server 200.

In this case, a model learner 2340 of the server 200 may performfunctions of the data trainer 1310 shown in FIG. 19 . The model learner2340 of the server 200 may learn a standard about which data is used todetermine the object recognition situation and how to determine theobject recognition situation by using the data. The model learner 2340may obtain data to be used for learning, apply the obtained data to thedata recognition model that will be described below, and learn astandard for determining the object recognition situation.

Also, the recognition result provider 1320-4 of the autonomous drivingdevice 100 may determine the object recognition situation by applyingthe data selected by the recognition data selector 1320-3 to the datarecognition model generated by the server 200. For example, therecognition result provider 1320-4 may transmit the data selected by therecognition data selector 1320-3 to the server 200, and the server 200may request the recognition model to determine the object recognitionsituation by applying the data selected by the recognition data selector1320-3 to the recognition model. The recognition result provider 1320-4may receive, from the server 200, information about the objectrecognition situation determined by the server 200.

Alternatively, the recognition result provider 1320-4 of the autonomousdriving device 100 may receive, from the server 200, the recognitionmodel generated by the server 200, and may determine the objectrecognition situation by using the received recognition model. In thiscase, the recognition result provider 1320-4 of the autonomous drivingdevice 100 may determine the object recognition situation by applyingthe data selected by the recognition data selector 1320-3 to the datarecognition model received from the server 200.

A method according to an embodiment may be embodied as program commandsexecutable by various computer means and may be recorded on acomputer-readable recording medium. The computer-readable recordingmedium may include program commands, data files, data structures, andthe like separately or in combinations. The program commands to berecorded on the computer-readable recording medium may be speciallydesigned and configured for embodiments of the present disclosure or maybe well-known to and be usable by one of ordinary skill in the art ofcomputer software. Examples of the computer-readable recording mediuminclude a magnetic medium such as a hard disk, a floppy disk, or amagnetic tape, an optical medium such as a compact disk read-only memory(CD-ROM) or a digital versatile disk (DVD), a magneto-optical mediumsuch as a floptical disk, and a hardware device specially configured tostore and execute program commands such as a ROM, a RAM, or a flashmemory. Examples of the program commands are advanced language codesthat may be executed by a computer by using an interpreter or the likeas well as machine language codes made by a compiler.

Some embodiments may be implemented as a recording medium includingcomputer-readable instructions such as a computer-executable programmodule. The computer-readable medium may be an arbitrary availablemedium accessible by a computer, and examples thereof include allvolatile and non-volatile media and separable and non-separable media.Further, examples of the computer-readable medium may include a computerstorage medium and a communication medium. Examples of the computerstorage medium include all volatile and non-volatile media and separableand non-separable media, which are implemented by an arbitrary method ortechnology, for storing information such as computer-readableinstructions, data structures, program modules, or other data. Thecommunication medium typically includes computer-readable instructions,data structures, program modules, other data of a modulated data signal,or other transmission mechanisms, and examples thereof include anarbitrary information transmission medium. Also, some embodiments may beimplemented as a computer program or a computer program productincluding computer-executable instructions such as a computer programexecuted by a computer.

While the present disclosure has been shown and described with referenceto various embodiments thereof, it will be understood by those skilledin the art that various changes in form and details may be made thereinwithout departing from the spirit and scope of the present disclosure asdefined by the appended claims and their equivalents.

The invention claimed is:
 1. A method, performed by an autonomousdriving device, of recognizing an object, the method comprising:obtaining a first red, green, blue (RGB) image by using a cameraarranged in the autonomous driving device; predicting at least one firstregion in which an object is unrecognizable in the first RGB image basedon brightness information of the first RGB image; determining at leastone second region in which an object exists, from among the at least onefirst region, based on object information obtained through a dynamicvision sensor (DVS) arranged in the autonomous driving device; obtainingan enhanced second RGB image by controlling photographic configurationinformation of the camera in relation to the at least one second region;and recognizing the object in the second RGB image.
 2. The method ofclaim 1, wherein the predicting of the at least one first region inwhich the object is unrecognizable comprises predicting a region inwhich brightness values are out of a threshold range in the first RGBimage as the at least one first region.
 3. The method of claim 1,wherein the predicting of the at least one first region in which theobject is unrecognizable comprises: analyzing the first RGB image toobtain a histogram of the first RGB image; determining whether anobject-unrecognizable region exists in the first RGB image by using thehistogram of the first RGB image; and when it is determined that theobject-unrecognizable region exists in the first RGB image, predictingthe at least one first region in the first RGB image based on thebrightness information of the first RGB image.
 4. The method of claim 1,wherein the predicting of the at least one first region in which theobject is unrecognizable comprises: determining whether anobject-unrecognizable region exists in the first RGB image by using afirst artificial intelligence model that has learned a plurality of RGBimages; and when it is determined that the object-unrecognizable regionexists in the first RGB image, predicting the at least one first regionin the first RGB image by using the first artificial intelligence model.5. The method of claim 1, wherein the object information comprises atleast one of a dynamic vision sensor (DVS) image obtained by the dynamicvision sensor and position information of at least one object detectedfrom the DVS image.
 6. The method of claim 1, wherein the determining ofthe at least one second region in which the object exists, from amongthe at least one first region, comprises determining the at least onesecond region by applying a DVS image obtained by the dynamic visionsensor and the first RGB image to a second artificial intelligencemodel.
 7. The method of claim 1, wherein the obtaining of the second RGBcomprises controlling at least one of an exposure, a focus, and a whitebalance with respect to the at least one second region.
 8. The method ofclaim 1, wherein the obtaining of the second RGB comprises adjusting atleast one of a gain, an aperture, and an exposure time of the camera. 9.The method of claim 1, further comprising obtaining, when the second RGBimage is composed of a plurality of frames, position information of theautonomous driving device by tracking a feature included in the objectrecognized from each of the plurality of frames.
 10. The method of claim1, further comprising determining a route of the autonomous drivingdevice based on information about the recognized object.
 11. The methodof claim 1, further comprising: tracking the recognized object by usingthe camera; detecting a new object appearing around the autonomousdriving device by using the dynamic vision sensor; determining, inresponse to the new object being detected, a candidate region in which aprobability of recognizing the new object in a third RGB image obtainedthrough the camera is greater than a threshold value; and recognizingthe new object from the third RGB image by performing image processingon the candidate region.
 12. An autonomous driving device comprising: acamera; a dynamic vision sensor (DVS); and at least one processor,wherein the at least one processor is configured to: obtain a first red,green, blue (RGB) image by using the camera; predict at least one firstregion in which an object is unrecognizable in the first RGB image basedon brightness information of the first RGB image; determine at least onesecond region in which an object exists, from among the at least onefirst region, based on object information obtained through the dynamicvision sensor; obtain an enhanced second RGB image by controllingphotographing configuration information of the camera in relation to theat least one second region; and recognize the object in the second RGBimage.
 13. The autonomous driving device of claim 12, wherein the atleast one processor is further configured to predict a region in whichbrightness values are out of a threshold range in the first RGB image asthe at least one first region.
 14. The autonomous driving device ofclaim 12, wherein the at least one processor is further configured to:analyze the first RGB image to obtain a histogram of the first RGBimage; determine whether an object-unrecognizable region exists in thefirst RGB image by using the histogram of the first RGB image; and whenit is determined that the object-unrecognizable region exists in thefirst RGB image, predict the at least one first region in the first RGBimage based on the brightness information of the first RGB image. 15.The autonomous driving device of claim 12, wherein the at least oneprocessor comprises an artificial intelligence processor configured todetermine whether an object-unrecognizable region exists in the firstRGB image by using a first artificial intelligence model that haslearned a plurality of RGB images, and when it is determined that theobject-unrecognizable region exists in the first RGB image, predict theat least one first region in the first RGB image by using the firstartificial intelligence model.
 16. The autonomous driving device ofclaim 12, wherein the at least one processor is further configured tocontrol photographic configuration information of the camera byadjusting at least one of a gain, aperture, and exposure time of thecamera.
 17. The autonomous driving device of claim 12, wherein the atleast one processor is further configured to: track the recognizedobject by using the camera; detect, by using the dynamic vision sensor,a new object appearing around the autonomous driving device; determine,in response to the new object being detected, a candidate region inwhich a probability of recognizing the new object in a third RGB imageobtained through the camera is greater than a threshold value; andrecognize the new object from the third RGB image by performing imageprocessing on the candidate region.
 18. The autonomous driving device ofclaim 12, wherein the at least one processor is further configured toset a frame rate of the dynamic vision sensor to be equal to a framerate of the camera.
 19. The autonomous driving device of claim 12,further comprising at least one of an autonomous driving vehicle, anautonomous flying device, and an autonomous driving robot.
 20. Acomputer program product comprising a recording medium having recordedthereon a program for: obtaining a first red, green, blue (RGB) image byusing a camera; predicting at least one first region in which an objectis unrecognizable in the first RGB image based on brightness informationof the first RGB image; determining at least one second region in whichan object exists, from among the at least one first region, based onobject information obtained through a dynamic vision sensor (DVS);obtaining an enhanced second RGB image by controlling photographingconfiguration information of the camera in relation to the at least onesecond region; and recognizing the object in the second RGB image.